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ABSTRACT 


Voice over Internet Protocol (VoIP) is growing in popularity in the civilian and 
military communities due to its low cost and the management advantages it offers over 
traditional Public Switched Telephone Networks (PSTN) phone systems. Many military 
commands do not have the infrastructure or funding that is required to support the rapid 
expansion of multiple phone services at various locations throughout the world. VoIP 
offers a rapidly deployable alternative. A subjective study was designed to test the quality 
of Voice over Internet Protocol (VoIP) signals in a controlled and isolated multilevel 
secure network to which single level networks were attached. The experiment provided 
useful insights regarding VoIP testing with human subjects and its procedures can be 
repeated as the Monterey Security Architecture (MYSEA) project moves forward with 


the implementation and deployment of VoIP services in its multilevel testbed. 
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GLOSSARY 


ARP: Address Resolution Protocol is a core protocol in the Transmission Control 
Protocol/Internet Protocol (TCP/IP), and is used to obtain the physical address of a host 


or node. ! 


CONFIDENCE INTERVAL: The confidence interval is an estimate of a population 
parameter that consists of a range of values bound by statistics referred to as “upper” and 
“lower” confidence limits, within which the value of the parameter is expected to be 


located.2 


CONFIDENCE LEVEL: The confidence level is an expression that represents how often 
the true percentage of the population would choose an answer which lies within the 


confidence interval’s upper and lower limits.3 


ENCODING: The process of transforming information from one format into another 


format. + 


GATEWAY: A router that connects two networks and can perform protocol 


conversion.- 


IEC: International Electro-technical Commission is an international standards and 


conformity assessment body for all fields of electro-technology. 


1 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 


2 Creative Research Systems, The Survey System (URL); http://www/surveysystem.com/sscalc.htm, 
[Accessed: February 20, 2008]. 


3 Tbid. 


4 Charlie Kaufman, Radia Perlman, Mike Speciner, Network Security Private Communication in a 
Public World, Prentice Hall PTR, 1995. 


5 Mani Subramanian,, Network Management, Principles and Practice, Addison-Wesley, 2000. 


XV 


IPv4: Internet Protocol version 4 is the fourth iteration of the Internet Protocol and is the 


Current standard for IP addressing. It specifies 32-bit addresses composed of four octets.® 


IPv6: Internet Protocol version 6 (IPv6) is a newer standard Internet Protocol addressing 
that will replace the [Pv4. IPv6 uses a newer, more efficient header in its packets and 
allows for 128-bit source and destination IP addresses. The addition of longer addresses 


will allow for more addresses. / 


ISO: International Organization of Standardization is an international standard-setting 


body composed of representatives from various national standards organizations.°® 


ITU: International Telecommunications Union is a United Nations agency that regulates 
international telecommunications and provides developing countries with technical 


expertise and equipment to advance their technological bases.? 


ITU-T: International Telecommunication Union Telecommunications Standardization 
Sector 1s a permanent subdivision of the International Telecommunications Union (ITU) 
and is responsible for providing standards by studying, and operating telecommunication 


networks. !9 


MAC: Media Access Control address is a 12 character string that uniquely identifies a 


network node.!! 


6 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 
7 Ibid. 


8 Charlie Kaufman, Radia Perlman, Mike Speciner, Network Security Private Communication in a 
Public World, Prentice Hall PTR, 1995. 


9 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 


10 International Telecommunications Union Standardization Sector (ITU-T), General Information, 
http://www. itu.int/ITU-T/info/index.html, [Accessed: February 14, 2008]. 


11 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 


XV1 


MLS: Multilevel Security: A class of systems containing information at different 
classification levels. Access decisions are based on the subject’s security clearances, need 


to know and formal approval.!2 


MYSEA: The Monterey Security Architecture 1s a trusted, distributed environment 


enforcing multilevel security policies. 


NIPRNet: Unclassified but Sensitive Internet Protocol Router Network, (formerly called 
the Non-Classified Internet Protocol Router Network). NIPRNet is used to exchange 
unclassified but sensitive information between "internal" users and also provides user 


access to the Internet. 


PSTN: The Public Switched Telephone Network is the network of the world's public 


circuit-switched telephone networks. 


QoS: Quality of Service is the ability to guarantee a certain level of performance for a 


data flow.!3 


SIP: Session Initiation Protocol is an application-layer control (signaling) protocol for 
creating, modifying, and terminating sessions with one or more participants. These 
sessions include Internet telephone calls (VoIP), multimedia distribution, and multimedia 


conferences. !4 


SIPRNet: The SIPRNet (Secret [formerly Secure] Internet Protocol Router Network) is a 
system of interconnected computer networks used by the U.S. Department of Defense 


and the U.S. Department of State to transmit classified information (up to and including 


12 Shon Harris, CISSP All-in-One Guide Third Edition, McGraw-Hill, 2005. 
13 Ibid. 


14 J. Rosenbury, The Internet Engineering Task Force (IETF), Request For Comments (RFC) 3261, 
The Internet Society, 2002, http://www. ieft.org/rfc/rfc3261.txt, [Accessed: February 14, 2008]. 


XVII 


information classified SECRET) in an identification and authentication environment that 


is isolated from the NIPRNet through a combination of logical and physical measures. 


SSO: Single Sign-On (SSO) is a method of access control that enables a user to 


authenticate once and gain access to the resources of multiple software systems. !° 


TCP/IP: Transmission Control Protocol/Internet Protocol is a suite of networking 


protocols that provides the foundation for data exchange across the Internet.!© 


TPE: Trusted Path Extension is a device responsible for providing a secure interface for 


user interaction with selected MYSEA server security functions. !7 


VoIP: Voice over Internet Protocol is a protocol for providing telephone service over a 


packet-switched network running the TCP/IP protocol suite.!® 


WTSA: World Telecommunication Standardization Assembly defines general policy and 


adopts working methods and procedures for the ITU-T. 


XML: The Extensible Markup Language (XML) 1s a general-purpose markup language. 
It is used on the World Wide Web in the context of the Hyper-Text Transfer Protocol. 1° 


15 Shon Harris, CISSP All-in-One Guide Third Edition, McGraw-Hill, 2005, 149-151. 
16 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 
17 Thuy D. Nguyen, Timothy E. Levin, Cynthia E. Irvine, MYSEA Testbed, 2004. 

18 Tamara Dean, Network + Guide to Networks, Thomson Course Technology, 2006. 


19 E. Whitehead, The Internet Engineering Task Force (ETF), Request For Comments (RFC) 2376, 
The Internet Society, 1998, <http://www.ieft.org/rfc/rfc2376.txt>, [Accessed: February 14, 2008]. 
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I. INTRODUCTION 


Voice over Internet Protocol (VoIP) is a growing technology that has been 
adopted and used by various components of the commercial sector. As with most of 
today’s advanced technology, the military is adopting this new digital communication 
concept, with some of the military services planning to implement VoIP in ground and 
sea deployments to provide integrated communications at all levels of the command 


structure29, 


The primary focus of utilizing VoIP technology is to provide voice 
communication down to the lowest level within the command. This technology will 
permit data exchange between separate commands without the need to deploy legacy 
Public Switched Telephone Networks (PSTN) equipment or infrastructures regardless of 
the location. When a command is deployed and interconnected to the main information 
grid, members of the command are able to communicate with one another, as well as with 


senior authorities as needed. 2! 


As the term Voice over IP implies, it is a PSTN replacement that enables 
connectivity to go beyond just speech communication. VoIP can be used for 
automatically sending additional information in the form of chat, video streaming, 


application files, and other web-based application sharing features. 


Even though the immediate deployment of VoIP in battle conditions is more 
spectacular and draws attention, VoIP’s beneficial contribution to permanent installations 
such as naval bases or naval air stations must not be downplayed. It is easy to see why 


VoIP should be considered for wide deployment, especially considering the cost savings 


20 Maryann Lawlor, Jeff Hawk, Henry S. Kenyon, Panelists Explore Network Centricity’s Many 
Facets, SIGNAL AFCEAS; International Journal, August 2005, 
http://www.afcea.org/signal/articles/templates/SIGNAL_Article_Temlate.asp’?articleid=10118&zoneid=8, 
[Accessed: March 21, 2007]. 


21 CMDR Jeffrey W. Eggers, Networks and Knowing, Armed Forces Journal, February 2008. 
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compared to traditional public switched telephone networks (PSTN). The installation cost 
is also decreased since the existing network infrastructure is used and manpower skills 


needed for administration are reduced22 


The real evolution in military communications will come from the combined use 
of VoIP within multilevel secure and non-multilevel secure networks. A deployed 
command will not need to carry copper/fiber cable, to enable communications. All 
commands can be interconnected without physical cabling linking them, and all service 
personnel can communicate seamlessly. First, the man-hours and skills needed to deploy 
and effectively administer the network are minimal, which is an important factor when 
the available manpower is limited and time is crucial. Second, it allows for further 
network expansion when additional troops arrive and the need for communication 


between networks is urgent?>. 


A. THESIS OBJECTIVE 


This thesis explores the quality of Voice over Internet Protocol (VoIP), when 
configured within a multilevel secure network. Subjective testing has been used to 
compare the results obtained in four network configurations, two multilevel secure and 
two non-multilevel secure networks. 

The International Telecommunications Standardization Sector (ITU-T) P.862.3,74 
was used as the standard in the subjective testing of voice quality within VoIP signals in a 
controlled and isolated multilevel network to which networks. 

The Monterey Secure Architecture, MYSEA, was the test bed for the two 
multilevel secure networks. The two non-multilevel secure networks had similar 
configurations and all networks had VoIP software, Skype, installed. 

Human test subjects were solicited from the staff and students of the Naval 


Postgraduate School to participate in the experiments. Each test subject was briefed and 


22 Karen E. Thuermer, Gearing up for Networx, Military Information Technology, Vol 11, Issue 8, 
September 25, 2007. 


23 B. Sklar, Digital Communications: Fundamental and Applications, Prentice Hall, Upper Saddle 
River, 2001. 


24 ITU-T Recommendation P.862 (2001) — Amendment 2. 
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instructed to listen to sound files on each network configuration and complete a separate 
ten-item questionnaire regarding each sound file. Each questionnaire was graded and 
tabulated using a Mean Opinion Score (MOS), which were used for single and multi 
attribute analysis. 

The hypothesis examined in this thesis is, “The quality of voice transmission over 
Internet Protocol networks in a multilevel secure network is comparable to a non- 


multilevel secure network.” 
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I. BACKGROUND 


A. OVERVIEW 


This chapter presents background information pertaining to this thesis study. The 
focus 1s the qualitative assessment of Voice over Internet Protocol (VoIP) within the 
multilevel secure network of the MYSEA (Monterey Security Architecture). The 
assessment method used is based on the testing guidelines and standardization of the 


International Telecommunications Union Standardization Sector (ITU-T). 


Although VoIP is an alternative to public switched telephone network (PSTN) 
services with respect to cost and flexibility, the quality of its output can determine 
whether, and under what conditions, VoIP will be acceptable to its end users when 
operated within a multilevel secure environment. This thesis will answer the following 
question: What is the quality of voice transmission over IP networks in a multilevel 
secure environment when compared to the same service in a non-multilevel secure 


environment? 


B. INTERNATIONAL TELECOMMUNICATIONS UNION 


The United Nations refers to the International Telecommunication Union (ITU) 
for specialized information and standards in the field of telecommunications. The ITU 
Telecommunication Standardization Sector (ITU-T) is a permanent subdivision of ITU. It 
is the ITU-T division’s responsibility to study technical configurations, operating 
procedures and resolve questions along with issuing recommendations on each proposed 
issue and to provide a technical viewpoint on standardizing telecommunications on a 
worldwide basis. Another related organization, The World ‘Telecommunication 
Standardization Assembly (WTSA), meets every four years, and establishes all of the 
topics for study by the ITU-T study groups. The ITU-T will then, produce 
recommendations on each of these topics. In some areas of information technology which 
fall within ITU-T's area of responsibility, standards are prepared by a collaborative 
foundation including the International Organization of Standardization (ISO) and The 
International Electro-technical Commission (IEC). 
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The International Telecommunications Union Standardization Sector (ITU-T) is 
the source of the standards for all test configurations and implementations used for this 
thesis study. The ITU’s standards date back to 1865, the date of formation of the 
International Telegraph Union. The ITU-T has been an intergovernmental public-private 
partnership organization since its inception. It became a specialized agency under the 
United Nations in 1947. It has a membership of 191 countries and over 700 public and 
private sector companies, as well as international and regional telecommunication 
entities.2° It was renamed in 1993 to its current title: International Telecommunications 
Union Standardization Sector (ITU-T). Its mission is to ensure efficient and timely 
creation of standards covering all fields of telecommunications on a worldwide basis, as 
well as defining bookkeeping principles for international telecommunications services. 
ITU-T’s use increased in the personal computer industry in the early 1980’s, often 
endorsing “de facto standards” as the technology was under development. The ITU-T 
developed a streamlined process that minimized the time between initial proposals and 
the final approval. Sometimes the time frame is as short as three months. The ITU-T 
technical work is managed by multiple study groups. Those involved in the study groups 


are experts in telecommunications from all over the world. 


C. MYSEA 


The Monterey Security Enhanced Architecture (MYSEA) was created at the 
Naval Postgraduate School to provide a trusted distributed operating environment that 
enforces multi-domain security policies, and supports commercial off the shelf (COTS) 
applications. MYSEA provides an isolated networking environment for the enforcement 
of stringent mandatory security policies. The architecture consists of a network that 
operates with high assurance MYSEA servers, low assurance MYSEA clients, legacy 
single level networks, and dynamic security services. MYSEA supports a secure trusted 
path for communications between the user and the trusted operating system. 

The trusted path of the MYSEA system is enabled by a Trusted Path Extension 
(TPE) device which creates a protected channel between itself and the MYSEA server. 


25 International Telecommunication Union. (2005). Telecommunication indicators Handbook. 
Geneva: ITU. <http://www.itu.int/ITU-D/ct/material/handbook.pdf>, [Accessed: January 16, 2007]. 


6 


The assurance properties of the TPE device ensure that the trusted path cannot be 
undermined by malware on any of the client workstations. Architecturally, the Trusted 
Path Extension provides the client’s only access to the MYSEA network for each logged 
on user and at the user’s sensitivity level. 
The MYSEA test bed continues to evolve as a growing technology that includes: 
a. Assured authentication and trusted paths to MLS services, (e.g., Email, Web- 
based Distributed Authoring and Versioning (DAV), etc.). 
b. Application of high assurance system-security technology to the integration of 
commercial and legacy components. 
c. Centralized security management. 
d. Integration of high assurance MLS with existing sensitive networks (e.g., 
SIPRNET, etc.). 
e. High assurance trusted communication channel techniques for managing 
access to various classified networks. 
f. Use of open architectures and standards. 
g. Experimentation on the use of an extensible markup language (XML) tags as 
security markings. 
h. Design of secure single sign-on (SSO) across multiple MLS servers. 
i. High assurance module authentication and verification. 


j. Secure collaborative information sharing. 


In order to maintain a strict and stable life cycle process within the MYSEA test 
bed, only authorized personnel are granted access to the multilevel networks and only 


secure hardware components and software releases are installed.?° 


D. VOIP 


Voice over Internet Protocol (VoIP) was first introduced in 1995 by Vocal Tec 
Inc., a telecommunication company that had launched a multimedia PC-based product 


which allowed users to speak through a PC microphone and listen to the conversation 


26 T. D. Nguyen, T. E. Levin , and C. E. Irvine, "MYSEA Test bed", Proceedings from the 6th IEEE 
Systems, Man and Cybernetics Information Assurance Workshop, West Point, NY, June 2005, 438-439. 
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from the PC’s speakers. This product was referred as the Internet Phone. VoIP uses 
Internet Protocols as the data transmission vehicle. A VoIP system digitizes voice using 
an audio codec, divides the digitized voice into packets, and sends the packets over an IP 
network to the intended destination. All packets are routed through the network so they 
will travel the same path. Unlike a public switched telephone network (PSTN) or legacy 
phone call, no dedicated circuit is ever created for a VoIP call. The exact process required 
to set up a VoIP call is dependent on the VoIP protocol. Two types of protocols are 
necessary to complete a VoIP call: 

1. Signaling, this has the function of establishing a session between the callers. 

2. Media transport, which specifies the rules and formats of the actual voice 


packets. 


For PSTN systems, a phone number is used to locate a phone via the switching 
network, whereas, a phone number in VoIP can be a regular PSTN phone number, an IP 
address, or an alias destination assignment. The “phone number” ultimately is translated 
to a 32-bit or 128-bit IP address, which is dependent on whether [Pv4 or IPv6 is used. 
Every VoIP signaling protocol must be capable of providing address resolution in order 
for each call to reach its destination. There are four general VoIP communication modes 


that are dependent upon this address resolution protocol (ARP): 


1 Phone-to-Phone, 

2 Phone-to-PC 

ai PC-to-Phone 

4 PC-to-PC 

Voice transmission is carried by both PSTN and IP networks under the first three 
modes, (Phone-to-Phone, Phone-to-PC, and PC-to-Phone). A VoIP service provider that 
interconnects the PSTN and VoIP networks is needed for the first three modes when a 
call originates from a PSTN network and arrives at a VoIP network or vice versa. Voice 
travels exclusively across the IP network in the fourth mode (PC-to-PC). 

The VoIP technology is made up of five distinct services: 

1. Signaling 

2: Encoding 


BF Transport 
4. Session Initiation Protocol 


>; Gateway Control 


A signaling VoIP protocol establishes and manages a connection between the 
endpoints when a call is made; this service requires the use of the VoIP signaling 
protocol. Session Initiation Protocol (SIP) is used as the media transport protocol and is 
used for setting up and breaking down voice calls. A primary objective in using SIP is to 
provide a connection and call setup protocol for [IP-based communications that can 
support a superset of the call processing functions and features present in the public 
switched telephone network (PSTN). These features permit telephone-like operations 
such as: dialing a number, enabling a telephone to ring, providing ring tones or a busy 
signal. Implementation and terminology are different in the SIP world but to the end-user, 


the behavior is similar to that of PSTN. 


SIP is a peer-to-peer protocol, where it requires only a simple core network with 
intelligence signals distributed throughout the network. Although many other VoIP 
signaling protocols exist, SIP is characterized as having its roots in the IP community 
rather than the telecom industry. When the conversation takes place, voice has to be 
encoded before it is transmitted over the IP network. The encoded voice packets will then 
be transported via the IP network to the destination. A gateway may be needed to convert 


voice into another format suitable for the receiving network.?/ 


27 Uyless D. Black, Voice over IP, 2™ Edition, Upper Saddle River, NJ, Prentice Hall, 2002, 330. 
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Hl. EXPERIMENTAL DESIGN 


This chapter describes the test methodology used to evaluate the quality of the 
audio content transmitted using VoIP in different network architectures. These 
architectures included a typical infrastructure and the MYSEA infrastructure. 

Testing was conducted on a dedicated test bed in the basement of Glasgow East 
and was based on a subjective approach using human test subjects. Each test subject 
answered a ten item questionnaire after listening to a sound file. This process was 
repeated for each of four different network configurations. All test data were collected 
from each test subject and retained for analysis. 

An initial experiment (Experiment I) revealed a potential bias in one of the four 
sound files. The initial testing was conducted with 61 test subjects presented with of four 
separate network configurations, where they listened to four separate sound files (sound 
files A, B, C, and D). However, each sound file remained at the same test station 
throughout the entire testing process. A review of the results indicated that Station Two 
displayed significantly higher scores than the remaining three stations. This indicated the 
need for an additional experiment to identify any potential sound file bias. 

A second experiment (Experiment II) was administered. It consisted of one basic 
network configuration at each of two separate testing stations. Forty new test subjects 
were solicited to participate in this experiment. For four test groups, the same four sound 
files were played in different order, (e.g., Trial 1: ABCD; Trial 2: BCDA; Trial 3: 
CDAB; Trial 4: DABC). The results of the second experiment indicated that Sound File 
B, had higher scores than the other three sound files regardless of the order in which they 
were presented. 

The results of Experiment II revealed that Station Two in Experiment I had a 
sound file of higher quality. Sound File B words were more understandable and had an 
overall better quality than the other three files. With this bias revealed, the results from 
Station Two were not used and Station One was the baseline for comparison of Stations 


Three and Four in Experiment I. 
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A. EXPERIMENT I 


The objective of this experiment was to measure voice quality using standard 
sound files downloaded from the International Telecommunications Union 
Standardization Sector (ITU-T) website and transmitted over four separate test 
configurations using Skype software.2® Two test configurations included the MYSEA 
infrastructure for the multilevel secure network and two test configurations did not 
include the MYSEA infrastructure. The main objective of the tests described in the 
following subsections was to determine if VoIP voice quality changes between MLS and 


non-MLS networks. 


1. Subject Sampling 


Speech quality was determined as the result of a subjective test, where a pre- 
determined number of test subjects listen to and judge the test material over separate 
networks via a ten-item questionnaire. All test subjects were random volunteers from the 
students and staff associated with the Naval Postgraduate School. The sample size of 61 
test subjects were selected from a total population of 1500 staff and students.2? For a 
confidence level of 95% this sample size corresponds to a 12% confidence interval.29 

Subjects were solicited through on-campus announcements which were posted at 
various academic buildings throughout campus, email solicitations from academic 
advisors/professors and classmates. These methods were the basis of success in obtaining 


the required sample size for both experiments. 


2. Test Question Selection 


Testing methods were centered on the objective of defining the subjective 
performance assessment of the quality of speech of VoIP in four separate network 


configurations. The method was a “listening test’, where messages were presented 


28 ITU-T Recommendation P.862 (2001) — Amendment 2. 


29 Fred N. Kerlinger, Foundations of Behavioral Research, 3rd Edition, (CBS College Publishing, 
1986), 117-119. 


30 Creative Research Systems, The Survey System (URL); http://www/surveysystem.com/sscalc.htm, 
[Accessed: February 20, 2008]. 
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aurally to all test subjects. Each test subject was given a ten-item questionnaire to answer 
at the conclusion of each sound file played at its associated station. Each of the test 
subject’s opinions were rated on a cardinality rating scale of 0 to 5, with 5 being of 
greater quality. Appendix A shows how the subjective opinions were mapped to 
cardinality scores. The cardinality point system was not known to the subject. These 
results were measures of the perceived quality in four categories of perception: Listening- 
quality effort, Articulation, Acceptance, and Overall Impression. The listening scores 
made it possible to compare the effectiveness of different speech files over each of the 
four separate network configurations. The questionnaires used throughout both 


experiments (I and II) were the same and are included in Appendix A.?! 


3. Sound File Selection 


ITU-T Recommendation P.862.3 was used to provide a guide for estimating 
listening speech quality by using reference sound files. This recommendation provides 
the necessary information for obtaining a stable and reliable reference for the listener’s 
perception of speech quality. 

It was recommended that the speech material consist of simple, short and 
meaningful sentences. These sentences were chosen to be easy to understand and were 
constructed into sets of two short sentences with no obvious connection of meaning 
between the sentences in a set. Additionally, in accordance with P.830, it was 
recommended that a minimum of two female and two male sound files be used for 


testing.>2 


4. Controlling Station 


The experiment controller played the sample files for all test subjects prior to 


testing at each station for sound and accent familiarity. Additionally, he initiated all 


31 ITU-T Recommendation P.85, Telephone Transmission Quality Subjective Opinion Tests, June 
1994. 


32 ITU-T Recommendation P.862.3, Telephone Transmission Quality, Telephone Installations, Local 
Line Networks, November 2005. 
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telephone connections between the controlling station and each of the testing stations via 
Skype VoIP software. The experiment controller used the same equipment for 
Experiments, I and II. 
a. Equipment: Dell Desktop CPU (Dimension 8200); four individual ITU 
pre-recorded voice sound files, (two male, two female) played via RealPlayer 
Enterprise Player, Build: 6.0.11.1526, Copyright (c) 2004; (1) 4 ft, 3.5 mm 
stereo audio cable (Male/Male). 
b. Network: Dell Desktop CPU connected to the Skype server via the Naval 
Postgraduate School Internet (unclassified Internet). 
c. Setup: A standard 3.5mm stereo audio cable (Male/Male) was connected 
to the speaker output jack (back of PC) and to the microphone input jack (back 
of PC) to create a loopback circuitry in order for the sound file to be 
transmitted to the Skype Server and back to the connected client for each of 
the four station configurations. 
NOTE: This testing was targeted for single connectivity, therefore, no conference calling 
features were tested or evaluated. All calls were made to one specific station at a time and 


only one file was transmitted over that connection. 


5. Station One 


Equipment: Dell Inspiron Notebook CPU (5150); Plantronics PC Headset. 

b. Network: Dell Desktop CPU (Dimension 8200) physically connected to the 
NPS Internet using CAT 5 cabling. 

c. Setup: ITU pre-recorded wave file played and transmitted from the 
experiment controller’s desktop to the test subject’s headset using Skype 
software. 

d. Configuration: As shown in Figure | there was no intermediate MYSEA 
Server (direct connection to the Internet and the Skype Server). In this 
configuration, Station One is connected directly to the Internet, via NPS 
Internet, using Skype software installed on the Dell Inspiron Notebook. Prior 
to testing, the experiment controller ensured that the laptop was logged on and 


connected to the Internet. The experiment controller opened and activated the 
14 
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Skype account for “Non-MLS One” client. After logging onto the Skype 
Server, the experiment controller initiated the call from the Dell desktop via 
the Skype Server. After the call was initiated and received by the “Non-MLS 
One” client, connectivity was established. Each test subject listened to File 


“A” at Station One, by using a Plantronics PC Headset. 
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Figure 1. Station One 


Station Two 


Equipment: Dell Inspiron Notebook CPU (5150); Plantronics PC Headset. 
Network: Dell Desktop CPU (Dimension 8200) physically to the NPS Internet 
with the Dell Inspiron Notebook CPU (5150) connected to a Proxy Server 
(Dell Desktop CPU-Dimension 8200) then into the NPS Internet using CAT 5 
cabling. 
Setup: ITU pre-recorded wave file played and transmitted from _ the 
experiment controller’s desktop to the test subject’s headset using Skype 
software. 
Configuration: A proxy server is interposed between the test subject and the 
Internet, via NPS Internet. Figure 2 shows that Station Two is connected to 
the Internet using a proxy server, Dell Desktop CPU (Dimension 8200), and 
Skype software installed on the Dell Inspiron Notebook. Prior to testing, the 
15 


experiment controller ensured the proxy server, and laptop were energized and 
logged onto the Internet. The experiment controller opened and activated the 
Skype account for “Non-MLS Two” client. After logging onto to the Skype 
Server, the experiment controller initiated the call from the Dell desktop via 
the Skype Server. After the call was initiated and received by the “Non-MLS 
Two” client, connectivity was established. Each test subject listened to File 


“B” at Station Two, by using a Plantronics PC Headset. 
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Figure 2. Station Two 


7. Station Three 


a. Equipment: Dell Inspiron Notebook CPU (5150); Plantronics PC Headset. 

b. Network: Dell Desktop CPU (Dimension 8200) physically to the NPS 
Internet. The Dell Inspiron Notebook CPU (5150) is connected to a Trusted 
Path Extension (TPE) through a MYSEA Server (BAE XTS 400) then into the 
NPS Internet using CAT 5 cabling. 

c. Setup: ITU pre-recorded wave file played and transmitted from the 
experiment controller’s desktop to the test subject’s headset using Skype 


software. 
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d. Configuration: MYSEA Server and Trusted Path Extension (TPE) are routed 
between Station Three and the Internet via NPS Internet. Figure 3 shows that 
Station Three is connected to the Internet via MYSEA using Skype software 
installed on the Dell Inspiron Notebook. Prior to testing, the experiment 
controller ensured the MYSEA server, and the laptop were energized and 
logged onto the Internet. The experiment controller opened and activated the 
Skype account for “MLS Three” client. After logging onto to the Skype 
Server, the experiment controller initiated the call from the Dell desktop via 
the Skype Server. After the call was initiated and received by the “MLS 
Three” client, connectivity was established. Each test subject listened to File 


“C” at Station Three, by using a Plantronics PC Headset. 
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Figure 3. Station Three 
8. Station Four 


a. Equipment: Dell Inspiron Notebook CPU (5150); Plantronics PC Headset. 

b. Network: Dell Desktop CPU (Dimension 8200) physically to the NPS 
Internet. The Dell Inspiron Notebook CPU (5150) is connected to a Trusted 
Path Extension (TPE), Dynamic Security Services Servers (Dell Dimension 
$200), and the MYSEA Server (BAE XTS 400) and then into the NPS 
Internet using CAT 5 cabling. 
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Setup: ITU pre-recorded wave file played and transmitted from the 
experiment controller’s desktop to the test subject’s headset using Skype 
software. 

Configuration: MYSEA, Dynamic Security Services (DSS)/Trusted Path 
Extension (TPE) Gateway Servers and Trusted Path Extension (TPE) are 
routed between Station Four and the Internet via NPS Internet. Figure 4 
indicates Station Four is connected to the Internet via MYSEA using Skype 
software installed on the Dell Inspiron Notebook. Prior to testing, the 
experiment controller ensured the MYSEA server and the laptop were 
energized and logged onto the Internet. The experiment controller opened and 
activated the Skype account for “MLS Four” client. After logging onto to the 
Skype Server, the experiment controller initiated the call from the Dell 
desktop via the Skype Server. After the call was initiated and received by the 


“MLS Four” client, connectivity was established. Each test subject listened to 


File “D” at Station Four, by using a Plantronics PC Headset. 
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Testing Procedures 


Speech Material: The ITU sound files consisted of short speech passages, 
chosen at random from the ITU database of sample sound files which were 
simple and self-contained in meaning. Each sound file (segment) has a 
duration of not less than 9 seconds and not more than 15 seconds, and 
consists of two “sentences”, which are separated by a pause of 
approximately 1 to 2 seconds. All sound files are of speech with a British 
accent, as recommended by ITU-T, and contained the phrases as shown in 
Appendix B. All sound files were played for each test subject over the 
Experiment Controller’s Dell desktop via the RealPlayer Enterprise Player 
and transmitted over a pair of external computer speakers prior to 
commencing the testing. This procedure was performed to familiarize the 
test subjects with the accent, gender, configuration and length of each 
sound file. Additionally, each test subject was allotted ample time to 
review the questionnaire and become prepared to answer each question as 
thoroughly and completely as possible. No time limits were imposed for 
completion of each of the questionnaires. 

Listening scale: The test subjects were instructed not to focus on the 
contents of the sentences but instead on the quality of sound. The rationale 
for this instruction, which was not conveyed to the participants, was that 
diversions regarding other aspects of the sound file could reduce the 
quality of the gathered information from the questionnaires, resulting in 
spurious assessment scores. 

Procedures: A ten-item questionnaire was distributed and available at each 
station. Each test subject was afforded ample time to answer the 
questionnaire after listening to the recording one time. Figure 5 shows that 
each test subject rotated to the next station after completing their assigned 


station. 
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d. Period: Experimental sessions lasted from 30-35 minutes for completion 
of all four stations. Four test subjects simultaneously occupied each of the 
four stations and rotated through all four stations. 

(1) Station transition: Approximately 1-2 minutes per test subject, per 
Station; total transition time between stations for all test 
subjects is 4-8 minutes. 

(2) Sound file transmission: approximately 1 minute per station; total 
sound file transmission time for all four stations 1s 4 minutes. 

(3) Questionnaire completion time: 5-6 minutes per subject for the 
questionnaire at each station, which resulted in a 20-25 minute 


period for completing four questionnaires. 


STATION ONE STATION TWO 


EACH SUBJECT ROTATES TO THE 
NEXT STATION AFTER COMPLETING 
THE PREVIOUS STATION 


STATION FOUR STATION THRE 


” 


Figure 5. Station Configuration & Rotation 
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B. EXPERIMENT II 


The objective of this experiment was to vary the order in which each of the four 
sound files were presented to the listener to determine if any sound files were of 
distinctly different quality than the others used in Experiment I. This experiment used the 
same standard sound files used in Experiment I. The sound files were transmitted to two 
separate test stations configured exactly the same using Skype software.?> The two test 
configurations did not include components of the MYSEA infrastructure, and used the 


same configuration as Station One, Experiment I. 


1. Subject Sampling 


The test subjects were to listen and judge the four sound files in a pre-determined 
order, not known by the test subject, over two identically configured stations over 
identically configured networks and complete four separate ten-item questionnaires for 
each of the sound files. 

All test subjects were random volunteers from the students and staff associated 
with the Naval Postgraduate School. The sample size of 40 test subjects was obtained 
from a total population of 1500 staff and students.3¢ For a confidence level of 95% this 
sample size corresponds to a 15% confidence interval.*° 

Subjects were solicited through on-campus announcements which were posted at 
various academic buildings, email solicitations from academic advisors/professors and 
word of mouth by classmates. These methods were the basis of success in obtaining the 


required sample size for both experiments. 


Zz Test Question Selection 


Testing methods were centered on the objective of obtaining a subjective 


performance assessment of the quality of sound of four separate sound files. The method 


33 ITU-T Recommendation P.862 (2001) — Amendment 2. 


34 Fred N. Kerlinger, Foundations of Behavioral Research, 3rd Edition, (CBS College Publishing, 
1986), 117-119. 


35 Creative Research Systems, The Survey System, 
http://www/surveysystem.com/sscalc.htm#ssneeded, [Accessed February 20, 2008] 
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was a “listening test” for sound file quality, where messages were presented aurally to all 
test subjects. Each test subject was given a ten-item questionnaire to answer after each 
sound file was played. Each of the test subject’s opinion scores were rated on a 
cardinality rating scale of 0 to 5, with 5 being of greater quality (Appendix A). The 
cardinality point system was not known to the subject. These results were measures of the 
perceived quality in four categories of perception: Listening-quality effort; Articulation; 
Acceptance and Overall Impression. These areas made it possible to compare the 
effectiveness of different speech files over each of the two network configurations. The 
questionnaires used throughout both experiments (I and II) were the same and are 


included in Appendix A.3¢ 


3. Sound File Selection 


ITU-T Recommendation P.862.3 was used to provide a guide for estimating 
listening speech quality by using reference sound files. This recommendation provides 
the necessary information for obtaining a stable and reliable reference for the listener’s 
perception of speech quality. 

It was recommended that the speech material consist of simple, short and 
meaningful sentences. These sentences were chosen to be easy to understand and were 
constructed into sets of two short sentences with no obvious connection of meaning 
between the sentences in a set. Additionally, in accordance with P.830, it was 
recommended that a minimum of two female and two male sound files be used for 


testing.>/ 


4. Controlling Station 


The experiment controller played the sample files for all test subjects prior to 


testing at each station for sound and accent familiarity. Additionally, he initiated all 


36 ITU-T Recommendation P.85, Telephone Transmission Quality Subjective Opinion Tests, June 
1994. 


37 ITU-T Recommendation P.862.3, Telephone Transmission Quality, Telephone Installations, Local 
Line Networks, November 2005. 
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telephone connections between the controlling station and each of the testing stations via 
Skype VoIP software. The experiment controller used the same equipment for both 
Experiments I and II. 

a. Equipment: Dell Desktop CPU (Dimension 8200); four individual ITU pre- 
recorded voice sound files, (two male, two female) played via RealPlayer 
Enterprise Player, Build: 6.0.11.1526, Copyright (c) 2004; (1) 4 ft, 3.5 mm 
stereo audio cable (Male/Male). 

b. Network: Dell Desktop CPU connected to the Skype server via Naval 
Postgraduate School Internet (unclassified Internet). 

c. Setup: A standard 3.5mm stereo audio cable (Male/Male) was connected to 
the speaker output jack (back of PC) and to the microphone input jack (back 
of PC) to create a loopback circuitry in order for the sound file to be 
transmitted to the Skype Server and back to the connected client. 

NOTE: This testing was targeted for single connectivity, therefore, no conference calling 
features were tested or evaluated. All calls were made to one specific station at a time and 


only one file was transmitted over that connection. 














PRE-RECORDED 
ITU SOUND FILES 





G 


Experiment 
Controller 


Test Subject 





Figure 6. Test Station One (TS1) and Test Station Two (TS2) 
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Test Station One (TS1) and Test Station Two (TS2) 


Equipment: Dell Inspiron Notebook CPU (5150); Plantronics PC Headset. 

. Network: Dell Desktop CPU (Dimension 8200) physically connected to the 
NPS Internet using CAT 5 cabling. 

Setup: ITU pre-recorded wave file played and transmitted from _ the 
experiment controller’s desktop to the test subject’s headset using Skype 
software. 

. Configuration: As shown in Figures 6, Test Stations One and Two are 
connected directly to the Internet, via NPS Internet, using Skype software 
installed on the Dell Inspiron Notebook. Prior to testing, the experiment 
controller ensured that the laptop was logged on and connected to the Internet. 
After the experiment controller logged both test stations onto to the Skype 
Server, he initiated the call from the Dell desktop via the Skype Server. After 
the call was initiated, the “TSI” and “TS2” clients obtained connectivity. 
Various test subjects listened to the four sound files at Test Station One, by 
using a Plantronics PC Headset. The order in which each file was transmitted 


and played will be discussed in the “Testing Procedures”. 


Testing Procedures 


Speech Material: The ITU sound files consisted of short speech passages, 
chosen at random from the ITU database of sample sound files and of which 
were simple and self-contained in meaning. Each sound file (segment) has a 
duration of not less than 9 seconds and not more than 15 seconds, and consists 
of two “sentences’’, which are separated by a pause of approximately | to 2 
seconds. All sound files are of speech with a British accent, as recommended 


by ITU-T, as shown in Appendix B. 


. Configuration: All sound files were played for each test subject over the 


Experiment Controller’s Dell desktop via the RealPlayer Enterprise Player 
and transmitted over a pair of external computer speakers prior to 


commencing the testing. This procedure was performed to familiarize the test 
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subjects with the accent, gender, configuration and length of each sound file. 
Additionally, each test subject was given ample time in reviewing the 
questionnaire in order to become prepared to answer the questions. No time 
limits were imposed on completing each of the questionnaires. 

c. Listening scale: The test subjects were instructed not to focus on the contents 
of the sentences but instead on the quality of sound. The rational for this 
instruction, which was not conveyed to the participants, was that diversions 
regarding other aspects of the sound file could reduce the quality of the 
gathered information from the questionnaires, resulting in spurious assessment 
scores. 

d. Procedures: Four (4) ten-item questionnaires were distributed and available at 
each of the two test stations. Each test subject was afforded ample time to 
answer the questionnaire after listening to each sound file. After completing 
the questionnaire for each sound file, the test subject indicated to the 
experiment controller that he/she was ready for the next sound file to be 
played. Sound files were played in a fixed order, as indicated in each “Trial” 
configuration. In Experiment II test subjects were tested on TSI and TS2, 
simultaneously. 

e. Period: Experimental sessions lasted from 25-30 minutes for completion of 
all four sound files. Two test subjects simultaneously occupied each of the 
two stations during playing of all four sound files for each Trial. 

f. Sound file transmission: approximately 1 minute per station; total sound file 
transmission time: | minute per station, total sound file transmission time for 
both stations were 2 minutes. 

g. Questionnaire completion time: 5-6 minutes per subject for the questionnaire 
at each station, which resulted in a 20-25 minute period for completing four 
questionnaires. 

This experiment subjectively tested voice quality of Voice over Internet Protocol 

on four network configurations, using human test subjects. The outcome of Experiment I 


produced results that were unanticipated and required further testing to validate the 
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quality of each of the sound files. The ITU-T sound files needed to be evaluated by 40 
test subjects not exposed to Experiment I. The conclusion of the Experiment II indicated 


that sound file “B” difference in quality from the other sound files. 
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IV. VOICE QUALITY ANALYSIS 


A. SCOPE OF ANALYSIS 
1. Introduction 


This chapter presents the results of testing used to evaluate the audio quality of 
VoIP in the Monterey Secure Architecture (MYSEA). Single Attribute analysis is the 
primary analysis technique. This form of analysis consists of sampling a single attribute, 
1e., voice quality, from the Mean Opinion Scores (MOS) derived from the ten-item 
questionnaire taken from subjective testing on four network configurations..3° 
Additionally, “Multi-attribute analysis” which is a comparison of MOS results of two 


questions was used in the analysis of Experiment I. 


In voice communication, audio quality usually dictates whether a listener’s 
experience is either good or bad. A numerical method of expressing voice quality is 
required. The method used here is called the Mean Opinion Score (MOS). In this method, 
voice quality is measured by having randomly selected test subjects rate the quality of 


test sentences transmitted over a communications circuit.2? 


The Mean Opinion Score (MOS) technique accurately determines subjective 
results*9, For this experiment, subjective testing required randomly selected test subjects 
to listen to pre-recorded standard files, and have them rate the audio quality according to 
their perception. To rate the audio quality, the test subjects were provided with a ten-item 


questionnaire. The questionnaire answers are presented in Appendixes C, and D. 


The ten-item questionnaire conformed to a Mean Opinion Score by rating each 
response with a numerical value ranging from 0 to 5. A mapping of question answers to 


numerical scores is shown in Appendix A. 


38 Jay L. Devore, Probability and Statistics For Engineering and the Sciences 6th Edition, 
Brooks/Cole, 2004, 713. 


39 ITU-T Recommendation P.862 (2001) — Amendment 2. 


40 William C. Hardy, “VoIP Service Quality: Measuring and Evaluating Packet-switched Voice’, 
McGraw-Hill Professional, 2003, 151. 
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At the conclusion of Experiment I, the results from one of the stations appeared 
anomalous. To identify any bias resulting from the quality of the sound file, all sound 
files used in Experiment I were tested for relative sound quality in Experiment II. This 


evaluation was performed using the same process as in Experiment I. 


2. Analysis of Experiment I 


Figures | through 6 in Chapter III present a pictorial representation of both 
experiments with four separate test configurations for Experiment I, and one 
configuration for Experiment II. The results were used for qualitative analysis of Voice 
over Internet Protocol (VoIP) in the MYSEA (Monterey Security Architecture) test bed 
using Skype, which is a commercial VoIP client and server software product. In 
Experiment I, each configuration is a separate station to be visited by each test subject. 
Each station added a component to the configuration for which voice quality in the test 
architecture was assessed. Station 1 and Station 2 measured the voice quality of VoIP in 
non-MYSEA networks, whereas, Stations 3 & 4 incorporated additional equipment 
(variables) from the MYSEA test bed network. A different sound file was selected to be 
transmitted to each of the test stations. (Appendix B identifies each sound file used at 
each specific station). 

Completed questionnaires were collected and entered into an Excel spreadsheet 
for analysis by station and question, (see Tables 3-14 in Appendix C). These test 
questions were assigned the MOS scores (see Appendix A) and averaged to provide an 
overall MOS rating for each of the stations. As Chart 1 of Appendix C shows, Station 
Two displays a higher MOS rating than that the other three stations. This result was 
unexplainable based on the network configuration of Station Two. 

Because Station One’s network configuration (see Experimental Design Chapter, 
Figure 1) was the simplest, it was expected to have presented the best MOS score. Station 
Two’s configuration, (see Experimental Design Chapter, Figure 2), included a “Proxy 
Server’ and should have resulted in the same or decreased quality; instead, it displayed 
increased quality. Stations Three and Four provided MOS scores similar in perceived 


quality to Station One. Since a given sound file was always presented at a given station, it 
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was suspected that the sound file played at Station Two might be the cause of the 


observed anomaly. Experiment II was designed to examine this possibility. 


a. Mult-attribute Analysis of Experiment I 


Multi-attribute data analysis methods complement the single-attribute 
analysis. This process has been applied to the results of Experiment I in comparisons of 
two similar questions.*! The goal of this analysis is to ascertain if two questions of 
similar meaning provide distinctive results when subjectively evaluating the quality of 


voice over internet protocol (VoIP) in a multilevel secure network. 


The following areas were assessed: 
1) Articulation: Perception on how clear the pronunciation of each 
spoken word (distinguish between words). 
2) Acceptance: Indicates if the voice quality was found to be 
acceptable for telephone conversations. 
3) Listening effort: The amount of effort required to understand the 
message. 
4) Overall impression: Overall assessment of word clarity, system 
connection and background noise when listening to a telephone message. 
Figures 4 through 15 (see Appendix E), are divided into four quadrants, 
representing the comparison between the two questions. Quadrant I (upper right corner), 
is represented with green color signifying “good” sound quality; Quadrants II, IV dower 
right and upper left, respectively) are yellow signifying “neutral” sound quality, and 
Quadrant III (lower left) is pink signifying “bad” or “poor” sound perception of the test 


subjects responses. 


This methodology of analysis reinforces the single-attribute analysis of the 
questions asked in reference to voice quality, sound quality and word clarity. Utilizing 
multi-attribute by comparing two similar questions and plotting the results into a 
quadratic graph, illustrates how the test subjects perceived quality of each of the sound 


files at the various stations. 


41 F. Murtagh and A. Heck, Multivariate Data Analysis, Kluwer, Dordrecht, 1987 
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a F Question 3 / Question 4 


The first multi-attribute analysis of two questions: 


Question 3: The voice on the recording was: 


O I Don’t remember or I’m not sure 
O Fading 

O Low Volume 

O Clear 


Question 4: The voice of the person on the recording sounded: Was it like an 


echo/shollow, fuzzy/unnatural or was it clear and understandable ? 


O I Don’t remember or I’m not sure 
O Fuzzy/Unnatural 

O Echo/Hollow 

O Clear and Understandable 


= 0 
=I 
=Z 
=3 


4. Question 9 / Question 1 


The second multi-attribute analysis of two questions: 


Question 9: How would you rate the quality of the sound of what you have just 


heard? 

O I Don’t remember or I’m not sure = 0 
O Bad =i 
O Poor = 2 
O Fair =3 
O Good =4 
O Excellent =) 


Question 1: Which of these four words comes closest to describing the quality of 


the connection during the recording? 


O Poor = 
O Fair = 2 
O Good =3 
O Excellent =4 


30 


>. Question 10 / Question 5 
The third and final multi-attribute analysis of two questions: 


Question 10: Were all of the words distinguishable ? 


O I Don’t remember or I’m not sure = 0 
O No, Not at all =-'| 
O No, Not very clear = 2 
O Fairly Clear = 
O Yes, Clear enough =4 
O Yes, Very Clear =) 


Question 5: Did you find certain words hard to understand? 


O I Don’t remember or I’m not sure = 0 
O Often —] 
O Occasionally = 
O Rarely = 3 
O Never = 4 
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Figure 7. Multi-Attribute Comparison of All Question Pairs 
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The Multi-attribute analysis supports the results of the single attribute analysis as 
indicated in Figure 7. Station Two received much higher quadrant one scores than any of 


the other stations. 


6. Analysis of Experiment II 


The objective of this experiment was to vary the order in which the four sound 
files were presented to a listener with a fixed network configuration to determine if one 
sound file was of better quality than the others. This experiment used the same ITU-T 
standard sound files as were used in Experiment I. The sound files were transmitted to 
two separate test stations configured exactly the same using Skype software.*2 Forty 
randomly and newly selected test subjects participated in Experiment II. Unlike 
Experiment I, each questionnaire distributed requested the gender of the Test Subject. 


This information was to help identify any listener/speaker gender bias. 


42 ITU-T Recommendation P.862 (2001) — Amendment 2. 
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Experiment II was designed to rotate the order in which each the sound files were 
presented: 

Trial 1: A, B, C, D 

Trial 2: B, C, D, A 

Trial 3: C, D, A, B 

Trial 4: D, A, B, C 

Appendix D, Chart 7 shows, each sound file tabulated by the test subject’s gender 
in order to determine if the anomalous scores in Experiment I, Station Two, could have 
come from gender biasing. Since the effect of test subject gender on the results were 
negligible, gender biasing was determined not to have been a factor in the anomalous 
outcome for Experiments I or H. Each question response was tabulated in Tables 16 
through 31. These responses were averaged and compared as displayed in Charts 9 
through 12 of Appendix D. 

Figure 8 shows that Station Two in Experiment I exhibited significantly higher 
scores than the other three stations. Additional testing was required to determine if sound 


file biasing was the cause of the anomalous reading. 
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Figure 9 confirms that the anomalous result for Experiment I resulted from sound 
file quality; in Experiment II. The higher sound quality of File B was consistent with the 
higher sound quality measured for Station Two in Experiment I; indicating that the 
increase was due to the quality of the sound file as perceived by test subjects and not 


because of the network configuration of Station Two. 
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V. CONCLUSION 


This thesis explored the quality of Voice over Internet Protocol (VoIP) when 
configured within a multilevel secure network and compared it to voice quality within a 
non-multilevel secure network. The standards of the Telecommunication Standardization 
Sector (ITU-T), P.862.3, were used for testing of voice quality within VoIP signals in a 
controlled and isolated multilevel network.43. The Monterey Secure Architecture, 
MYSEA, was the test bed multilevel secure network, where VoIP software, Skype, was 
used to transmit the sound files. This call processing software was used for all 
experiments and on all test stations. 

The test subjects consisted of staff and students from the Naval Postgraduate 
School. Each test subject was instructed on Skype’s operational capability and 
functionality. Additionally, they were instructed to listen to the four sound files and 
complete a ten-item questionnaire for each sound file which were assigned to a specific 
testing station. Experiment I was configured for the test subjects to evaluate the sound 
files at each of four network configurations. Each station questionnaire was graded and 
tabulated using a Mean Opinion Score (MOS), which was used for single and multi 
attribute analysis. Once the MOS was scored and plotted, an anomalous result appeared 
for Station Two. Since a particular sound file was always presented at a particular station, 
it was suspected that the sound file played at Station Two might be the cause of the 
observed anomaly. Experiment II was designed to examine the possibility of sound file 
bias. 

The objective of Experiment II was to determine if one sound file was of better 
quality than the others. This was achieved by varying the order in which the four sound 
files were presented to each listener within a fixed network configuration. Using the same 
ITU-T standard sound files as were used in Experiment I, all sound files were transmitted 
to two separate test stations configured exactly the same, as shown in Figure 6, using 


Skype software.*+ Forty test subjects participated in Experiment II. The questionnaires 


43 ITU-T Recommendation P.862 (2001) — Amendment 2. 
44 ITU-T Recommendation P.862 (2001) — Amendment 2. 
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for Experiment II requested the gender of each Test Subject. This information was to help 
identify any listener/speaker gender bias. Experiment II rotated the order in which each of 
the sound files was presented as outlined in the Experimental Design Chapter. 

Test subject gender showed negligible differences on the results of Experiment II. 
Thus test subject gender bias was dismissed as a factor in the anomalous outcome for 
Station Two in Experiment I. Experiment II results provided evidence that the sound file 
“B” had scores indicating a statistically significant anomalous outcome. 

All of Station Two results provided in Experiment I were disregarded and Station 
One was used as the baseline for the remaining testing and analysis. The experimental 
results indicate that voice quality in multilevel secure networks scored as “good” or 
“better” voice quality when compared to the same characteristics in a non-multilevel 


secure network. 
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APPENDIX A. QUESTIONNAIRE 


Appendix A lists the ten-item questionnaire administered to participants in both 


Experiments (I & II) after they listened to each of the sound files. 


Each question is assigned a point value for analysis computation. 
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QUESTIONS 
Which of these four 


words comes closest to 
describing the quality of 
the connection during 
the recording ? 

Did you have difficulty 


in listening over the 
connection 





The voice on _ the 
recording was: 





The voice of the person 


on the recording 
sounded: Was it like an 
echo/hollow, 

fuzzy/unnatural or was 
it clear and 


understandable ? 







Did you find certain 
words hard to 
understand ? 


How would you 
describe the effort you 


were required to make 
in order to understand 
the message? 


SS ee eee 3 4 


Table 1. Ten Item Questionnaire 


POOR FAIR GOOD EXCELLENT 
NO 
I DON'T 
REMEMBER I'M LOW 
NOT SURE FADING VOLUME CLEAR 
I DON'T FUZZY / CLEAR AND 
REMEMBER I'M | UNNATURA ECHO / UNDERSTAND 
NOT SURE LU HOLLOW -ABLE 
I DON'T 
REMEMBER I'M OCCASION 
NOT SURE ALLY RARELY 
ATTENTION 
NECESSARY; | COMPLETELY 
NO COMPREHEN 
NOT COMPRE- MODERAT | APPRECIABLE SIBLE; 
HENSIBLE WITH EFFORT E EFFORT EFFORT NO EFFORT 
ANY EFFORT REQUIRED | REQUIRED REQUIRED REQUIRED 
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Table 1. Ten Item Questionnaire (cont) 














QUESTIONS Po a a 

Tell me if you noticed any 
of these during your I DIDN'T 
session: A rushing or NOTICE ANY 
hissing sound; a frying NOISE OR 
and /or sizzling, FRYING ANYTHING 
crackling sound; or a I DON'T AND/OR OTHER THAN 
humming or buzzing | REMEMBER I'M | HUMMING, | SIZZLING, RUSHING, THE 

7 | sound? NOT SURE BUZZING | CRACKLING HISSING RECORDING 


Please try to remember 
the background noise in 
the area around your 
station (eé.g., noise from 
the air-conditioning unit, 
office equipment, or 


while you were involved 
the following categories I DON'T 
best describes your | REMEMBER I'M VERY 
S | experience? NOT SURE NOISY NOISY QUIET VERY QUIET 


other people _ talking) 
in the testing. Which of 
How would you rate the 
quality of the sound of I DON'T 
what you have just | REMEMBER I'M 
9 | heard? NOT SURE BAD POOR FAIR GOOD EXCELLENT 





I DON'T NO, NOT 
Were all of the words | REMEMBERIM | NO, NOT VERY YES, CLEAR YES, VERY 
10 | distinguishable ? NOT SURE AT ALL CLEAR FAIRLY CLEAR ENOUGH CLEAR 
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APPENDIX B. SOUND FILES 


The International Telecommunications Union Standardization Sector (ITU-T) provided 


sound files were used in end-to-end speech quality assessment of telephone networks. 


1. Sound file **A’’: British female voice (af1s03b2c17) with the words: 


“He took out his pipe and lit up. It was a separate bar.” 


2. Sound file **B’’: British male voice (am1s03b2c4) with the words: 


“He carried a bag of tennis balls. The scheme was plotted out.” 


3. Sound file **C’’: British female voice (af1sO1b2c16) with the words: 


“You were the perfect Hostess. Are you going to be nice to me?” 


4. Sound file **D’’: British male voice (am1s0O2b2c4) with the words: 


“There wasn’t a scrap of evidence. The jar was full of water.” 
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APPENDIX C. EXPERIMENT I DATA 


Appendix C shows questionnaire data from each of the 61 test subjects taken during Experiment I. 
Table 2. Station Average MOS Scores 


AVERAGE DATA COMPARISON 


eee aaa ae] ed ed Pe 
STATION TWO AVERAGE (S2) » 2.93 | 2.97 | 2.79 | 3.36 | 3.38 | 3.64 | 2.64 | 4.30 | 4.20 
poms [oat (2a a [aa aaa a a 


Table 2 is the average scores for each question for each station. See Appendix A for Mean Opinion Score (MOS) point assignment to 
questionnaire complete sentences. 
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AVERAGE OF ALL QUESITIONS 


3 ATION 1 3 TATION 2 5 TATIONS 5 TATION 4 





Chart 1. Station Comparison of All Questions 


Chart 1 compares the average score for all question for each station . 
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Chart 2. Stations and Question Comparison 


Chart 2 shows the comparison of each station by each individual question. The Mean Opinion Score value assigned is detailed in the 
information box in the upper right corner of Chart 2. 
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Station One Raw Data 


Table 3. 


TEST STATION 





One 


Test Subject 1 


Test Subject 2 
Test Subject 3 
Test Subject 4 
Test Subject 5 
Test Subject 6 
Test Subject 7 
Test Subject 8 
Test Subject 9 
Test Subject 10 
Test Subject 11 
Test Subject 12 
Test Subject 13 
Test Subject 14 
Test Subject 15 
Test Subject 16 
Test Subject 17 
Test Subject 18 
Test Subject 19 
Test Subject 20 





4 


pl 


4 


l 


gy) 


3 


3 


3 


3 
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Station One Raw Data (cont) 


Table 4. 


4 
4 
3 
5 
3 
3 


3 
2 
2 
2 
3 
ps 





Qa | @ | @ | @ | @ 
3 2 
3 1 


Q2 


1 
3 
3 

















4 
3 
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Table 5. Station One Raw Data (cont) 


TEST STATION 


One 
Test Subject 41 3 3 
Test Subject 42 3 4 
] 3 


| TestSubject43_ | 3) | 3) | 3) | 1 | 3) | 3 


. 









Oo | & | Go 
Oo | Go 
Uo | Go 
Uo | Go 


Test Subject 33 
Test Subject 54 


Test Subject 357 3 0 4 

| TestSubject58_ | 3) | 3) | 2 | 1 | 3 | 2 | 3 4 

| TestSubject59 | 2 | 3) | 3 | 3 | 2 | 3 | 3 p44 | 3 
Test Subject 60 1 0 3 

2 {| 3 | 2 


Tables 3, 4 and 5 show the responses from the 61 test subjects who listened to Sound File “A” at Station One. 
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STATION ONE COMPARISON 


AVERAGE 
SCORES (61 
TEST 
SUBJECTS) 



































> 6 
QUESTIONS 





Chart 3. Station One Average Question Responses 


Chart 3 shows the average scores of 61 test subjects on the ten item questionnaire at Station One. 
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Table 6. Station Two Raw Data 





TEST STATION . 
Two a | a | @ | @ | @ | @ | @ | Qw 
Test Subject 1 3 J 
Test Subject 4 4 3 3 3 4 4 4 2 
Test Subject 6 4 3 3 3 4 4 4 4 
Test Subject 7 2 3 3 p 3 3 l 3 
| TestSubjectto | 3 | 3 | 3 | 1 | 4 | 3 | 4 | 2 
| TestSubjectt1_ | 3 | 3 | 3 | o | 4 | 3 | 2 | a 
| TestSubject12_ | 4 | 3 | 3 | 3 | 4 | 4 | 4 | 4 


iN 


Test Subject 13 


3 
3 
4 
4 
| TestSubjecti4_ | 4 | 3 | 3 | 3 | 3 | 3 | 4 [| 2 | 
3 
3 
3 
3 
4 


Lo | & 
Ly | Go 
Ly | Go 
Co | Go | Go 
NO | Go 


Test Subject 15 


| TestSubjectio | 4 | 3 | 3 | 3 | 2 | 3 | 4 | 2 
| Test Subject17_ | 4 


Test Subject 19 


| TestSubject20 | 4 | 3 | 3 | 3 | 2 | 3 | 4 | 3 
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iN 
Uo | Uo 
ta" |G 
































Oo | Go | Go 
HWM [NY jo; HRB 


oe ee ee 





K 


iN 
ae 

aa 

oe 


“| 


oS) 


Minin jyhR HI IH IH nH LH PH IH SO Pn ni ns | HH 
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as 
Uo 
Uo 
Uo 
I) 
as 
Uo 


Table 7. Station Two Raw Data (cont) 








Two 


Test Subject 24 4 
3 


4 
| TestSubject25 | 3 | 3 | 3 | 3 | 3 | 4 | 4 | 2 
Test Subject 26 
3 p 4 
| 
| TestSubject29 | 3) | 1 Ewa 
Test Subject 30 4 3 
| 
4 p22 
3 | 3 | 4 | 2 
3 | 3 | 3 | 4a ft 4 ft 8 
4 4 3 
4 3 
| 2 
em 
| 2 


© 


0 


K 


oS) 
oS) 
£4 


K 
UO |bOo 
UO 


Lo | Go | Go 
Lo | Go | Go 
£4 4 








| Test Subject35_ | 4 
Test Subject 36 


2 
3 
4 
4 
2 
4 
4 
2 
3 
2 
3 
2 
2 
3 
3 
2 
pi 
pi 
3 


oS) 
- 
se 


MiB |N[OS |H [nay HR nH [nH Jo HR HR IHR Tn HR |H | nn | n | n 


3 tf 3 | 4 | 
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Table 8. Station Two Raw Data (cont) 


TEST STATION 





Two 






| TestSubject41 | 4 | 3 | 3 | 3 | 4 | 4 | 4 | 2 | 5 | 5 
- 

Test Subject 43 l l l ] 1 l 2 2 p) 2 
: 
5 
: 

Test Subject 47 4 3 3 3 2 4 4 3 4 4 
5 
5 
3 

Test Subject 31 4 3 3 3 4 3 4 2 ) 5 
4 
: 
5 

Test Subject 55 3 3 3 2 2 3 4 3 4 3 
4 
4 
5 
: 
4 
3 2 2 2 - 


Tables 6, 7, and 8 show the responses from the 61 test subjects who listened to Sound File “B” at Station Two. 
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AVERAGE 
SCORES (61 
TEST 


SUBJECTS} 





Chart 4. Station Two Average Question Responses 


Chart 4 shows the average scores of 61 test subjects on the ten item questionnaire at Station Two. 
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Table 9. Station Three Raw Data 


Test Subjects 1 
Test Subjects 2 
3 


Test Subjects 


Test Subjects 8 


Test Subjects 11 
Test Subjects 12 








3 3 3 3 4 
po tf tof tof te | 3 ft 3} 38 fF 2 
3 pi Z 3 4 3 3 3 
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Station Three Raw Data (cont) 


Table 10. 





Qs | Qo | Qo 


Qs 





Qs 


Qs | Qu 


Q2 


Qi 


TEST STATION 
Three 





3 3 i 3 2 3 2 ps 4 3 


21 


Test Subjects 
Test Subjects 


Test Subjects 23 


22 


24 


Test Subjects 


Test Subjects 25 


Test Subjects 26 


27 


Test Subjects 


Test Subjects 28 


Test Subjects 29 


Test Subjects 30 
Test Subjects 
Test Subjects 


31 


32 


Test Subjects 33 


Test Subjects 34 
Test Subjects 
Test Subjects 


35 


36 


Test Subjects 37 
Test Subjects 38 


Test Subjects 39 
Test Subjects 40 





Ss. 


Table 11. Station Three Raw Data (cont) 





TEST STATION El Dt é 
Three a | @ | @ | @ | @® | @wW | @® | ® | Ww | Qn 
2 
3 2 
4 
3 
| TestSubjects47_ | 3 | 3 | 3 | 3 | 1 | O | 4 2 
Test Subjects 48 4 3 3 3 1 l 4 2 2) ps 
| TestSubjects51_ | 2 | 1 | 3 | 2 | 1 | O | 3 | 2 | 3 | 2 
Test Subjects 54 3 3 3 l pi) 3 4 3 4 3 
3 3 J 3 
5 


_2 | 3/4 | 2 4 

3 

pl 0 
i 


I 
| TestSubjectson | 2) | 1 | i | bt 7 tot tof Oo fF 2 | 2 fF 2 


oS) 
oS) 





— 
NO | — 
NO | Go 


Test Subjects 60 y) 


pd 

NO 

NO} Oo] & 
oS) 


Tables 9, 10, and 11 show the responses from the 61 test subjects who listened to Sound File “C”’ at Station Three. 
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STATION THREE COMPARISON 

































































AVERAGE SCORES (61 
TEST SUBJECTS) 





















































Chart 5. Stations Three Average Question Responses 


Chart 5 shows the average scores of 61 test subjects on the ten item questionnaire at Station Three. 
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Station Four Raw Data 


Table 12. 


| Qr | Qs | ® | Qo 


é 
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Station Four Raw Data (cont) 


Table 13. 
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Table 14. Station Four Raw Data (cont) 





Tables 12, 13, and 14 show the responses from the 61 test subjects who listened to Sound File “D” at Station Four. 
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STATION FOUR COMPARISON 



























































AVERAGE 
SCORES (61 
TEST 
SUBJECTS) 





Chart 6. Station Four Average Question Responses 


Chart 6 shows the average scores of 61 test subjects on the ten item questionnaire at Station Four. 
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APPENDIX D. EXPERIMENT Il DATA 
Appendix D provides questionnaire data from each of the 40 test subjects taken during Experiment II. 


Table 15. Average questionnaire scores as a function of sound files. 


AVERAGE DATA COMPARISON 


see we ele wee 
2 Prag ers Oo ere re De 
sooo [am fan [an [as [an [aw lam [ew | em 
pores Lae [ae [ae ae | ae ean [on [a 


Table 15 is the average scores for each question for each sound file. See Appendix A for point assignment to questionnaire complete 
sentences. 
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OVERALL SOUND FILE COMPARISONS 


~ OVERALL AVERAGE 


~ FEMALE AVERAGE 
MALE AVERAGE 


FILE “A* FILE" B" FILE FILE “D" 





Chart 7. Average scores are shown as a function of the listener gender. 
Chart 7 Shows the average score for each sound file. 
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SOUND FILE 
COMPARISONS 














SOUND FILE "A" 
all 


SOUND FILE "B" 


PF, 
! 


/ = SOUND FILE "1" 


~ SOUND FILE "D" 


QUESTIONS 





Chart 8. All Sound File Comparison 
Chart 8 shows the average responses for each question for each sound file. 
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Table 16. Trial 1 Sound File ''A' Raw Data 






SOUND FILE | KVEY ESTIO! 
a ee SF,Qi SF,Q> SF,Qs 
oe | » | 3 
Trial 1-2 7 Pog 
Trial 1-3 3 
3 
3 





SFxQu 
s |: 


2 


oS) 
- 


1 {4 1 | oa | 2 | 2 | 2 
| *Trial 1-7 | 9) | 1 | 1 | 


3 
3 
A 
A 


WW |OO TOO [bo 
- 
- 

Ww JH [MR Jw 
- 
7 
- 
-|. 


—_— 


oS) 
LW |N 


oS) 

_— 
ae 
NO 
NW JO [uO 
£ IN Tf [kM 
Ww JMR Jw [NM 
WwW ;R Jo 
NW Jb [uO 


Trial 1-8 3 





Trial 1-9 ®) 1 


3 
triatt10 | 3 | 3 a |os | 2 fe fs 


Note: * Indicates Female Test Subject Response 


we) 
iN 


Sound file **A’’: British female voice (af1sO3b2c17) with the words: “He took out his pipe and lit up. It was a separate bar.” 
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Table 17. Trial 1 Sound File ''B" Raw Data 





ae aa Sra =r fies [sf 

VB! SFpQi | SFsQ2_ | SFsQ3 | SFaQ4 | SF8Qs | SFaQ6 | SF8Q7_ | SFaQs | SFaQo | SFaQuo 
— |. 

Trial 1-2 SSE 
Trial 13 
Trial 4 mo s | 
Trial 8 
cr ee ed s | 
“Trial 1-7 pep 
Trial 18 pa fs [is | a | s s | os 


mare |. |stats 1.1. w|i 


Note: * Indicates Female Test Subject Response 


Lo & 
aaa 
aad 
& & 
— Nn 


i) = - 
iN Nn Nn 
Se) Nn -& 


Sound file **B’’: British male voice (am1s03b2c4) with the words: “He carried a bag of tennis balls. The scheme was plotted out.” 
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Table 18. Trial 1 Sound File ''C'' Raw Data 
























SOUND FILE 
me" SFcQ; SFcQ> 









SFcQ3 | SFcQs | SFcQs | SFcQo 





SFcQ7 | SFcQs 





cra fs fs | 

pew 
3 

-can 


Trial 1-5 


















] 


Trial 1-8 


Note: * Indicates Female Test Subject Response 





2 

2 

4 

1 
vs 

&, 
Sound file **C”’: British female voice (af1sO1b2c16) with the words: “You were the perfect Hostess. Are you going to be nice to me?” 
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Table 19. Trial 1 Sound File ''D'"' Raw Data 


BEE 87,0, | sx0, | sos | sro0. | smp0: | si0s | srr | St02 |sMe05 
"Dp" SFpQi | SFpQ2 | SFpQ3 | SFpQ4 | SFpQs | SFpQc6 | SFpnQ7 | SFpQs_ | SFpQo 
Trial 1-1 3 3 3 3 3 3 4 2 4 

Trial 1-2 
titt3 | > | 3 | 3 | o | 2 | 2 | «4 | 2 | 3 | 3 
Trial 1-4 ! 





Trial 1-5 3 3 3 3 2 2 4 2 5 3 
iat6 | > | 1 | 1 | 1 | a | o | 3 | 2 | a | 


“Trial 1-7 3 3 3 3 3 2 0 3 3 
mitts fo; [os | > | i | 3s | 3 fo | 3 | a | 
; 
; 


Note: * Indicates Female Test Subject Response 
Sound file *D’’: British male voice (am1s02b2c4) with the words: “There wasn’t a scrap of evidence. The jar was full of water.” 


Tables 16, 17, 18, and 19 show the results from Trial | and the responses from the 40 test subjects who listened to all sound files in the 
order of: “A, B, C, D”. 
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TRIAL 1 
AVERAGE COMPARISON 


= OVERALL 
AVERAGE 


| © FEMALE 
AVERAGE 


MALE 
AVERAGE 








FILE"A" FILE"B" FILE"C"  FILE"D* 
SOUND FILES 





Chart 9. Trial 1 Sound File Comparison 


Chart 9 shows the responses of the test subjects by gender. 
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Table 20. Trial 2 Sound File ''B" Raw Data 













SURVEY QUESTIONS 

SOUND FILE ae 

mB SFBQo SFRQi0 
* Trial 2-3 3 3 
Trial 2-4 
Trial 25 
Trial 2-6 ! 
*Trial 2-7 3 3 3 4 3 4 2 5 4 
UNE ae 3 3 3 4 4 4 2 5 5 
Trial 2-9 A 2 5 5 


iN 
ad 


Trial 2-10 


Note: * Indicates Female Test Subject Response 


Sound file **B’’: British male voice (am1s03b2c4) with the words: “He carried a bag of tennis balls. The scheme was plotted out.” 
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Table 21. Trial 2 Sound File “C’’ Raw Data 


nat ce Sik eh sth sath lh sto 
65 SFcQi | SFcQ2. | SFcQ3 | SFcQs =| SFcQs) | SFcQ6 | SFcQ7 =| SFcQs | SFcQo | SFcQio 





| * Trial 2-3 - ~ , ~ - ~ ~ ~ a 


a 
| Trial 2-4 3 3 iS 3 i 4 = Ld i 3 La — 
Trial 2-5 
Pewee | sts ts fs 1lo)+ 2! . 
mar fe es 


Trial 2-8 


| triat29 | 4 iG 4 is 3 - i 4 - - — 
aaa] 3 I 1 3 3 2 2 2 _ 


Note: * Indicates Female Test Subject Response 


Sound file **C’’: British female voice (af1sO1b2c16) with the words: “You were the perfect Hostess. Are you going to be nice to me?” 


a2 


Table 22. Trial 2 Sound File “D’’ Raw Data 





OND (sro, [sues |stuos |sruo. [snes [sre [sues sruo [snes [sro 
"Dp" SFpQi_ | SFnQ2_ | SFnQ3 | SFnQs_ | SFnQs_ | SFpQ6_ | SFpQ7_ | SFpQs_ | SFpQo | SFnQuo 
nae 2 ofa fa [ia 


Trial 2-2 


*Trial 2-3 





Trial 2-4 


“Trial 2-7 


Trial 2-10 — 5 — 1 — — — 0 — — ~ 


Note: * Indicates Female Test Subject Response 
Sound file *D’’: British male voice (am1s02b2c4) with the words: “There wasn’t a scrap of evidence. The jar was full of water.” 
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Table 23. Trial 2 Sound File “A”? Raw Data 


ache wa fo [a fi |i lo fo [a foo [a 
ae SFaQi | SFaQo | SFaQ3 | SFaQa | SFaQs | SFaQo | SFaQ7 | SFaQs | SFaQo | SFaQio 
Trial 2-1 

Trial 2-2 SEES 
"Trial 2-3 
Trial 2-4 


Teil 2s 
mize | oe fos fo | Pe Pl Pe Pl fe 
“Trial 27 pe fe pe pe 


a 
| Triatzto | 2 | 3 | ot ft Ps Ps Pe 


Note: * Indicates Female Test Subject Response 





Sound file **A’’: British female voice (af1sO3b2c17) with the words: “He took out his pipe and lit up. It was a separate bar.” 


Tables 20, 21, 22 and 23 show the results of Trial 2 and the responses from the 40 test subjects who listened to all sound files in the 
order of “B, C, D, A’. 
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TRIAL2 
AVERAGE COMPARISON 


| sve 
AVERAGE 

ay ra 
880) = AVERAGE 


Ey MALE 


AVERAGE 


Fai 


FILE"B" FILE"C" FILE"D" FILE "A" 
SOUND FILES 





Chart 10. Trial 2 Sound File Comparison 


Chart 10 shows the responses of the test subjects by gender: 
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Table 24. Trial 3 Sound File “C’’ Raw Data 


SOUND FILE 
"Cc 


Trial 3-1 


rere 


Trial 3-3 


| 
*Trial 3-5 

Sse [fs . 

, 

| 

| 


| 3 | 2 | 3 | 3 | 4 | 3 








Note: * Indicates Female Test Subject Responses 


Sound file **C’’: British female voice (af1sO1b2c16) with the words: “You were the perfect Hostess. Are you going to be nice to me?” 
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Table 25. Trial 3 Sound File “D’’ Raw Data 










SOUND FILE 
"yy! 





Note: * Indicates Female Test Subject Responses 


Sound file *D’’: British male voice (am1s02b2c4) with the words: “There wasn’t a scrap of evidence. The jar was full of water.” 


7T7/ 


Table 26. Trial 3 Sound File “A” Raw Data 


or 
oe SFaQi | SFaQ2 | SFaQ3 


Trial 3-1 


na [sf [a 


* 1 = 


Trial 3-7 


wave |. [a [a | 


* Trial 3-9 


vo | s 1a 1a , 








Note: * Indicates Female Test Subject Responses 


Sound file **A’’: British female voice (af1sO3b2c17) with the words: “He took out his pipe and lit up. It was a separate bar.” 
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Table 27. Trial 3 Sound File “‘B” Raw Data 





SOUND FILE 
"B" SFpQ; | SFaQ2 | SFaQ3 | SFaQa | SFsnQs_ | SFsnQo | SFsnQ7 | SFaQs | SFaQo | SFaQio 


Trial 3-7 





Note: * Indicates Female Test Subject Responses 


Sound file **B’’: British male voice (am1s03b2c4) with the words: “He carried a bag of tennis balls. The scheme was plotted out.” 


Tables 24, 25, 26 and 27 show the results of Trial 3 and the responses from the 40 test subjects who listened to all sound files in the 
order of “C, D, A, B 


79 


TRIAL3 
AVERAGE COMPARISON 


© OVERALL 
AVERAGE 
EMALE AVERAGE 


MALE AVERAGE 








FILE"C"  FILE"D" FILE"A"  FILE"B" 
SOUND FILES 





Chart 11. Trial 3 Sound File Comparison 


Chart 11 shows the responses of the test subjects by gender. 
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Table 28. Trial 4 Sound File “D’’ Raw Data 








SOUND FILE 


"D' sro. sr. 
1 fio |r | 2 fis 


Trial 4-3 3 l 2 







2 
Trial 4-6 


Trial 4-7 


* Trial 4-8 


“Trial 4-9 
Trial 4-10 


Note: * Indicates Female Test Subject Responses 





Sound file *D’’: British male voice (am1s02b2c4) with the words: “There wasn’t a scrap of evidence. The jar was full of water.” 


81 


Table 29. Trial 4 Sound File ‘A’? Raw Data 











"A" a “a 7 a SFaQ6 | SFaQ7) | SFaQg | SFaQo | SFaQio 
= arr 
* 

Teil 42 SSS sts ofa |. 
minds | 5 | i | 3 | | | 3 | ls | | 2 | 
mints | 5 | s |i fs | s | > | > | a | 
* Trial 4-5 2 9) 2 3 3 
wes [> [stata to” to to | 
wins? |g | os | os | | sts |] 3 
“Trial 48 ee fats ; 





* Trial 4-9 


Note: * Indicates Female Test Subject Responses 


Sound file **A’’: British female voice (af1sO3b2c17) with the words: “He took out his pipe and lit up. It was a separate bar.” 
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Table 30. Trial 4 Sound File “‘B’”’ Raw Data 







SOUND FILE 
"B iB} 


*Trial 4-1 


“Trial 4-2 





Trial 4-4 


% 1 ie 


* 7 a 
Trial 4-9 4 3 3 3 4 4 4 2 5 5 
Trial 4-10 


Note: * Indicates Female Test Subject Responses 





Sound file **B’’: British male voice (am1s03b2c4) with the words: “He carried a bag of tennis balls. The scheme was plotted out.” 
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Table 31. Trial 4 Sound File “C’’ Raw Data 





SOUND FILE "C" SFcQi | SFcQs 
; 
; 






SFcQ¢ 


sr. 


ps Pp 2 


: 


Nw JM TM [hd 


Trial 4-7 4 a 2 2: a 


* Trial 4-8 3 3 2) 3 
sway [5 | s | sp 


3 
Trial 4-10 3 





Note: * Indicates Female Test Subject Responses 
Sound file **C’’: British female voice (af1sO1b2c16) with the words: “You were the perfect Hostess. Are you going to be nice to me?” 


Tables 28, 29, 30 and 31 show the results of Trial 3 and the responses from the 40 test subjects who listened to all sound files in the 
order of “D, A, B, C. 
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TRIAL4 












AVERAGE COMPARISON —__ 
© FEMALE 


& ‘4 AVERAGE 
lew i © MALE AVERAGE 


4.00 
3.00 7 
2.00 
1.00 





0.00 





FILE"D" —FILE"A" FILE "B" FILE" G" 
8 OUND FILES 


Chart 12. Trial 4 Sound File Comparison 


Chart 12 shows the responses of the test subjects by gender 
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MULTI-ATTRIBUTE ANALYSIS 


APPENDIX E 


Appendix E shows the comparison of two similar questions which illustrates how the test subjects perceived quality of each of the 


sound files at the various stations. 


Question 3 & 4 Raw Data 


Table 32. 


& 
~) 
O 
LL 
ra 
0 
d 
- 
VI 
LLJ 
LL] 
a 
ae 
- 
a 
0 
d 
- 
VI 
O 
3 
E 
ra 
0 
d 
- 
VI 


STATION ONE 























Test 
Subject 
TS-11 
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Question 3 & 4 Raw Data (cont) 


Table 33. 


STATION THREE STATION FOUR 


STATION TWO 


STATION ONE 


Test 


= 


ce 


1 


2 


a 


5 


TS-26 


3 


1 


T3-27 





15-28 


Ta-43) 


TS-30 


OO ee eee 


3 


Ts-38 


Th 


TS-39 


T5-40 
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Question 3 & 4 Raw Data (cont) 


Table 34. 


STATION THREE STATION FOUR 


STATION TWO 


STATION ONE 


eh 
O] m 
= 


Subject 


- 
: 


Ta-a1 





Tables 32, 33 and 34 show the responses from the 61 test subjects at each Station for questions 3 and 4. 
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Questions 3 & 4 Station One 


Figure 10. 
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PO ECEIBEL 
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Questions 3 & 4 Station Two 


Figure 11. 
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CLEAR 


LOW VOLUME 
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= 
os 
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Figure 12. Questions 3 & 4 Station Three 
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Figure 13. Questions 3 & 4 Station Four 


Figures 10, 11, 12 and 13 show the percentage of responses from the 61 test subjects within each quadrant. 
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Questions 9 & 1 Raw Data 


Table 35. 





STATION FOUR 


STATION THREE 


O 
s 
- 
a 
© 
q 
- 
Ww) 





STATION ONE 







































































Test 
Subject : 
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Questions 9 & 1 Raw Data (cont) 


Table 36. 








e 
_ 
O 
LL 
= 
© 
c 
~ 
Vv) 
Ww 
Wu 
ce 
L 
— 
ra 
= 
: 
Ww 
O 
s 
-_ 
= 
© 
a 
-_ 
Vv) 


STATION ONE 


























. 
AE | 


Cl 
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Questions 9 & 1 Raw Data (cont) 


Table 37. 


STATION FOUR 






































STATION THREE 











0 
3 
- 
= 
0 
r 
j= 
vf) 


STATION ONE 





Subject 
Ts-41 


Tables 35, 36and 37 show the responses from the 61 test subjects at each Station for questions 9 and 1. 
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Questions 9 & 1 Station One 


Figure 14. 
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Questions 9 & 1 Station Two 


Figure 15. 
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Questions 9 & 1 Station Three 


Figure 16. 
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Figure 17. Questions 9 & 1 Station Four 


Figures 14, 15, 16 and 17 show the percentage of responses from the 61 test subjects within each quadrant. 


100 


Questions 10 & 5 Raw Data 


Table 38. 





SlATION FOUR 


Ss lLA'TION THREE 


. 
i 
am 
a 
‘s 
— 
=| 
=f 
am 
wa 


SlLATION ONE 











TEST 
SsUBIEWTS 


1 
| 
. 
o] 
af 
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Questions 10 & 5 Raw Data (cont) 


Table 39. 





SlA TION FOUR 


Ocal ca] cal emf at] en} tl ent cal st] oonf ot] ca] caf t] ml fot] cal 
wa 























Ss TATION THREE 


















































() 


— 
am 
a 
af 
| 
fase 
“1, 
am 
A 


SlLATION ONE 
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Questions 10 & 5 Raw Data (cont.) 


Table 40. 


eTPATION POUR 


eA TIOM THREE 


0 
te 
= 
a 
: 
= 
4 
bi 
we 


ePATION ONE 


EERE EREEERRERREEEE 
Selah el eal ale a) | | | sl 
a ji RORYORUGVGR 
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Tables 38, 39 and 40 show the responses from the 61 test subjects at each Station for questions 10 and 5. 


AMDA'SHA| HVITD ‘SHA 


ATHIVA |AMZALON'ONLVLON 'ONIMHQWHWGE 
LiNOd] 





Questions 10 & 5 Station One 


Figure 18. 
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Questions 10 & 5 Station Two 


Figure 19. 
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Questions 10 & 5 Station Three 


Figure 20. 
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YES, VERY 
CLEAR 





YES, CLEAR 
ENOUGH 


oa 
= 
i 
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— 
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Figure 21. Questions 10 & 5 Station Four 


Figures 18, 19, 20 and 21 show the percentage of responses from the 61 test subjects within each quadrant. 
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APPENDIX F THESIS PROPOSAL AND IRB APPROVAL 
FORMS 


Appendix F consists of the approved thesis proposal and institutional review board (IRB) 
forms. 


————-" 


YEMORANTDUM 
Dae: Muy 29, 2007 
From: 1.4 DR CO’ M. ADARIS, JR. 


Vo Prowear: Of Ger, Indormulion Aysiems, and Ciperations, Curriculum 336 


Ww 
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